Clean up Prometheus labels scraped from proxy #633
Conversation
The Prometheus scrape config collects from Conduit proxies, and maps Kubernetes labels to Prometheus labels, appending "k8s_". This change keeps the resultant Prometheus labels consistent with their source Kubernetes labels. For example: "deployment" and "pod_template_hash". Signed-off-by: Andrew Seigner <siggy@buoyant.io>
| # special case k8s' "job" label, to not interfere with prometheus' "job" | ||
| # label | ||
| # __meta_kubernetes_pod_label_conduit_io_proxy_job=foo => | ||
| # k8s_job=foo |
There was a problem hiding this comment.
I've been on both sides of the fence on this, but the fact that we have to prefix job to avoid conflicts makes me kinda think we should go back to prefixing all resources for consistency...
There was a problem hiding this comment.
Agree it's unfortunate that k8s_job ends up being a special case. I think in general it will make our code slightly more readable. An example I just ran into:
#627 (comment)
...where we're referring to pod-template-hash when talking to Kubernetes and then k8s_pod_template_hash immediately after when querying Prometheus.
There was a problem hiding this comment.
Also I'll note that Googling around yields a direct k8s->prom label mapping to be pretty standard, and what folks would naturally expect when querying Prometheus for Kubernetes data:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L266-L267
| # __meta_kubernetes_pod_label_conduit_io_proxy_deployment=foo => | ||
| # k8s_deployment=foo | ||
| # deployment=foo | ||
| - action: labelmap |
There was a problem hiding this comment.
what does this action do when replacement is not specified?
There was a problem hiding this comment.
per https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config, replacement defaults to the matched group $1:
[ replacement: <string> | default = $1 ]
The Prometheus scrape config collects from Conduit proxies, and maps
Kubernetes labels to Prometheus labels, appending "k8s_".
This change keeps the resultant Prometheus labels consistent with their
source Kubernetes labels. For example: "deployment" and
"pod_template_hash".
Signed-off-by: Andrew Seigner siggy@buoyant.io